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(54) Method and apparatus for a color scannerless range imaging system 



(57) A color scannerless range imaging system 
comprises four component systems, including an illumi- 
nation system, an image capture device, an optical as- 
sembly and a controller for interconnecting and control- 
ling the component systems. The illumination system 
separately illuminates a scene with modulated and un- 
modulated illumination, while the image capture device 
is positioned in an optical path of the reflected illumina- 
tion from the scene for capturing a plurality of images 
thereof, including (a) at least one image of the reflected 
modulated illumination, whereby the modulation of the 
reflected modulated illumination incorporates a phase 
delay corresponding to the distance of objects in the 
scene from the range imaging system, and (b) at least 
one color image of the reflected unmodulated illumina- 
tion. The optical assembly includes a first optical system 
for imaging a color image of unmodulated illumination 
from the scene and a second optical system including 
a modulating stage for modulating the reflected modu- 
lated illumination from the scene and generating an in- 
terference image representative of range information of 
the scene. The controller selectively places each of the 
optical systems In the optical path of the image capture 
device whereby both range information and color infor- 
mation of the scene are captured. 
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Description 

FIELD OF THE INVENTION 

5 [0001] The invention relates generally to the field of three-dimensional image capture and In particular to the capture 
of three-dimensional image information with a scannerless range imaging system. 

BACKGROUND OF THE INVENTION 

10 [0002] Standard image capture systems will capture images, such as photographic images, that are two-dimensional 
representations of the three-dimensional world. In such systems, projective geometry best models the process of trans- 
fomiing the three-dimensional real world into the two-dimensional images. In particular, much of the information that 
is lost in the transformation is in the distance between the camera and image points in the real world. Methods and 
processes have been proposed to retrieve or record this information. Some methods, such as one based on a scanner 

'5 from Cyberware, Inc., use a laser to scan across a scene. Variations in the reflected light are used to estimate the 
range to the object. However, these methods require the subject to be within 2 meters of the camera and are typically 
slow. Stereo imaging is a common example of another process, which is fast on capture but requires solving the 
"correspondence problem", that can be difficult and limit the number of pixels having range data, due to limited feature 
points. 

20 [0003] Another method described in U.S. Patent 4,935,616 uses a scanner-less laser range imaging system to illu- 
minate a scene object with an amplitude modulated laser source. In this patent, the image capture portion of the system 
includes an image intensifier (in particular, a micro-channel plate) that is connected to modulate the optical signal 
reflected from the scene object. The distance- related phase shift of the intensity modulation reflected from the scene 
object can be determined by capturing two images. A first image is captured without modulating the optical signal and 
^5 a second image Is captured with the received optical signal modulated by the micro-channel plate in phase with the 
same amplitude modulated frequency as used to modulate the laser source. Both captured images are registered 
spatially, and the relationship between them is a function of the range to the object in the scene. Once the phase shift 
has been established, range to the object can be recovered. The second image may be taken by phase shifting either 
the illumination modulation or the image intensifier modulation. After the Images are acquired they are processed on 
30 a pixel-by-pixel basis to ascertain the range from the camera to the object for each pixel. 

[0004] The preferred method of estimating the range in the '61 6 patent uses a pair of captured images, one image 
with a destructive interference caused by modulating the image intensifier, and the other image with the Image intensifier 
set at a constant voltage. However, a more stable estimation method uses a series of at least three images, each with 
modulation applied to the image intensifier as described In commonly assigned copending application Serial No. [our 
Docket No. 78510MGBJ, entitled "Method and Apparatus for Scannerless Range Image Capture Using Photographic 
Film" and filed in the names of Lawrence Allen Ray and Timothy P. Mathers. In that application, the distinguishing 
feature of each image Is that the phase of the image intensifier modulation for each image is unique relative to mod- 
ulation of the illuminator. If a series of n images are to be collected, then the pref en-ed arrangement is for successive 
images to have a phase shift of ^ radians (where n is the number of images) from the phase of the previous image. 
40 However, this specific shift is not required, albeit the phase shifts need to be unique. The resultant set of images is 
referred to as an image bundle. The range at a pixel location is estimated by selecting the intensity of the pixel at that 
location In each image of the bundle and perfomning a best fit of a sine wave of one period through the points. The 
phase of the resulting best-fitted sine wave is then used to estimate the range to the object based upon the wave- 
length of the illumination frequency. 
45 [0005] A drawback of methods based on the "616 patent is that color information is lost. Unfortunately for color 
applications, an image intensifier operates by converting photonic energy into a stream of electrons, amplifying the 
energy of the electrons and then converting the electrons back into photonic energy via a phosphor plate. One conse- 
quence of these conversions is that color information is lost. Since color is a useful property of images for many ap- 
plk^ations, a means of acquiring the color information that is registered along with the range information is extremely 
so desirable. 

[0006] The system described in the '616 patent may be implemented in relation to a normal camera system; in 
partteular, a standard camera system may be converted into a range capture system by modifying its optical system. 
The camera may operate as either as a digital camera or a camera utilizing film. In the case of a film based system 
there are some other requirements that need to be met. These requirements and means for satisfying them are de- 
scribed In the aforementioned copending application Serial No. [our Docket No. 78510MGBJ, entitled "Method and 
Apparatus for Scannertess Range Image Capture Using Photographic Film". As mentioned above, the drawback of 
such a camera system is its Inability to capture a color image. What is needed is a convenient camera system that can 
capture ranging infonmation without sacrifteing the color information that It would otherwise capture. 
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SUMMARY OF THE INVENTION 

[0007] It Is an object of the invention to provide a scannerless range imaging system that is capable of capturing 
both range images and color images of a scene. 

5 [0008] The present invention is directed to overcoming one or more of the problems set forth above. Briefly summa- 
rized, according to one aspect of the present invention, a color scannerless range imaging system for generating 
images of a scene comprises an Hlumination system for controllably illuminating the scene with modulated illumination, 
whereby some of the modulated illumination is reflected from objects in the scene; an image capture device positioned 
in an optical path of the reflected illumination from the scene for capturing a plurality of images thereof, including (a) 

10 at feast one image of the reflected modulated illumination, whereby the modulation of the reflected modulated illumi- 
nation incorporates a phase delay corresponding to the distance of objects In the scene from the range Imaging system, 
and (b) at least one color image of reflected unmodulated illumination; and means for storing the plurality of images 
as a bundle of associated images. 

[0009] According to another aspect of the invention, a camera system for color scanneriess range Imaging comprises 
IS four component systems, including an illumination system, an image capture device, an optical assembly and a con- 
troller for Interconnecting and controlling the component systems. The illumination system separately illuminates a 
scene with modulated Illumination, while the image capture device is positioned in an optical path of the reflected 
illumination from the scene for capturing a plurality of images thereof. Including (a) at least one Image of the reflected 
modulated illumination, whereby the modulation of the reflected modulated Illumination incorporates a phase delay 
20 corresponding to the distance of objects In the scene from the range imaging system, and (b) at least one color image 
of reflected unmodulated illumination. The optical assembly includes a first optical system for imaging a color image 
of unmodulated illumination from the scene and a second optical system including a modulating stage for modulating 
the reflected modulated illumination from the scene and generating an interference image representative of range 
infomnation of the scene. The controller selectively places each of the optical systems in the optical path of the image 
25 capture device whereby both range Information and color information of the scene are captured. 

[0010] In a further aspect, the illumination system additionally produces unmodulated illumination for Illuminating the 
scene and producing at least in part the reflected unmodulated illumination captured by the image capture device as 
the color image. 

[0011] The advantage of the present invention is that it provides a means of obtaining a color image along with range 
30 information for each point on the image. Besides using a scannerless range image capture method, which is rapid and 
operative over a longer range than other methods, the invention includes a dual illuminating system and means for 
interchanging optical elements rapidly and simply so that both range and color Information may be efficiently captured 
under the best Illuminating conditions for each capture. The ability to accomplish this is provided in part by having the 
range capture system embodied as a camera attachment, e.g., on a lens turret, that is optically coupled with the Image 
35 capture device. 

[0012] These and other aspects, objects, features and advantages of the present invention will be more cleariy 
understood and appreciated from a review of the following detailed description of the prefen-ed embodiments and 
appended claims, and by reference to the accompanying drawings. 

40 BRIEF DESCRIPTION OF THE DRAWINGS 

[0013] FIG. 1 shows the main components of a color scanneriess range Imaging system in accordance with the 
invention. 

[0014] FIG. 2 Is a diagram illustrating an image bundle and related data captured by the system shown in Figure 1 . 
45 [0015] FIG. 3 is a diagram showing more detail of the image capture device shown In Figure 1 . 
[0016] FIG. 4 is a diagram showing more detail of the illumination system shown in Figure 1. 
[0017] FIG. 5 Is a diagram showing more detail of the optical assembly shown In Rgure 1 . 
[0018] FIG. 6 is a diagram showing more detail of the controller shown In Figure 1 . 

[0019] FIG. 7 is a diagram of the steps involved in processing Image and range information in accordance with the 
50 invention. 

[0020] FIG. 8 is a block diagram of a range imaging system which can be used to capture a bundle of images. 
[0021] FIG. 9 is a perspective diagram of a computer system for implementing certain programs associated with the 
present invention. 

55 DETAILED DESCRIPTION OF THE INVENTION 

[0022] Because range imaging devices employing laser Illuminators and capture devices Including Image intensif iers 
and electronic sensors are well known, the present description will be directed in particular to elements fomiing part 
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of, or cooperating more directly with, apparatus in accordance with the present invention. Elements not specifically 
shown or described herein may be selected from those known In the art. Certain aspects of the embodiments to be 
described may be provided in software. Given the system as described In the following materials, all such software 
implementation needed for practice of the invention is conventional and within the ordinary skill in such arts. 

5 [0023] It is helpful to first review the principles and techniques involved in scannerless range Imaging, particularly in 
connection with the capture of image bundles in accordance with the aforementioned copending application Serial No. 
[our Docket No. 7851 OMGB], entitled "Method and Apparatus for Scanneriess Range Image Capture Using Photo- 
graphic Film". Accordingly, referring first to Figure 8, an range imaging system 1 0is shown as a laser radar that is used 
to illuminate a scene 12 and then to capture an image bundle comprising a minimum of three images of the scene 12. 

10 An illuminator 1 4 emits a beam of electromagnetic radiation whose frequency Is controlled by a modulator 1 6. Typically 
the illuminator 14 Is a laser device which includes an optical diffuser in order to effect a wide-field Illumination. It is 
preferable that the modulating laser is an IR source. This Is for eye-safety issues and to operate In the spectral region 
of maximal response by the capture system. The modulator 16 provides an amplitude varying sinusoidal modulation. 
The modulated illumination source is modeled by: 

15 

L(t) = + Tisi" (2«>'0 (Eq. 1) 

where [l^ is the mean illumination, r\ is the modulus of the illumination source, and X is the modulation frequency applied 
to the Illuminator 14. The modulation frequency Is sufficiently high (e.g., 10 MHz) to attain sufficiently accurate range 
estimates. The output beam 18 Is directed toward the scene 12 and a reflected beam 20 is directed back toward a 
receiving section 22. As is well known, the reflected beam 20 is a delayed version of the transmitted output beam 18, 
with the amount of phase delay being a function of the distance of the scene 12 from the range Imaging system. The 
reflected beam 20 is sensed by a photocathode 24, which converts its amplitude variations Into a modulated electron 
stream that strikes an image Intensifier 26. Since the photocathode 24 does not separately process the colors In the 
scene, the electron stream created at this point is essentially monochromatic, i.e., the color information is lost at this 
time. The output of the image intensifier 26 is modeled by: 

30 M(t) =\Lm+ ysm {2KXti (Eq. 2) 

where >s the mean intensification, y is the modulus of the intensification and X Is the modulation frequency applied 
to the intensifier 26. The purpose of the image intensifier is not only to intensify the image, but also to act as a modulating 
shutter. Accordingly, the image intensifier 26 is connected to the modulator 16, such that the electron stream strikes 
the intensifier 26 and is modulated by a modulating signal from the modulator 1 6. The modulated electron stream is 
then amplified through secondary emission by a mrcrochannel plate 30. The intensified electron stream bombards a 
phosphor screen 32, which converts the energy into a visible light image. The intensified light image signal Is captured 
by a capture mechanism 34, such as a charge-coupled device (CCD). The captured image signal is applied to a range 
processor 36 to detennine the phase delay at each point in the scene. The phase delay term co of an object at a range 
p meters is given by: 
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o = ?£^mod 2k (Eq. 3) 

where c is the velocity of light in a vacuum. Consequently, the reflected light at this point Is modeled by: 

R(t) = + sin (2jcX/ + <o) (Eq. 4) 



50 



where k is the modulus of illumination reflected from the object. The pixel response P at this point Is an Integration of 
the reflected light and the effect of the intensification: 



55 ^ = ^(^)A^(')^' = Tt-^KTqe cos(^) (Eq. 5) 
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[0024] In the range imaging systenn disclosed in the aforementioned '616 patent, a reference image is captured 
during which time the micro-channel plate is not modulated, but rather kept at a mean response. In that case, equation 
(5) fundamentally is unchanged, though M(t) is now simply a constant ^|^. The range is estimated for each pixel by 
recovering the phase term as a function of the value of the pixel in the reference image and the phase Image. There 
s are several reasons why this approach is not robust. Included in this is the fact that the analysis depends upon con- 
tinuous values. The range estimation is based upon the portion of the phase image relative to the reference image. 
For digital systems the relative quantization of the phase image to the reference Image decreases as the response of 
the reference image decreases. The system is also somewhat noise sensitive. 

[0025] A robust approach which overcomes the limitations of the method proposed In the '616 patent is described 
*o in the aforementioned Serial Number (Docket 78510), which Is Incorporated herein by reference. Instead of collecting 
a phase image and a reference image, the improved approach collects at least three phase Images (referred to as an 
image bundle). In the previous approach, the intenslfier26 and the laser Illuminator 1 4 were phase locked. The Improved 
approach shifts the phase of the intensif ier 26 relative to the phase of the illuminator 1 4, and each of the phase images 
has a distinct phase offset. For this purpose, the range processor 36 is suitably connected to control the phase offset 
f 5 of the modulator 1 6, as well as the average illumination level and such other capture functions as may be necessary. 
If the image intensif ier 26 (or laser illuminator 1 4) is phase shifted by 6/, the pixel response from equation (5) becomes: 



Pj = 2\i^\if^fi + Kicy cos(o> + 6/) (Eq. 6) 

20 

[0026] It is desired to extract the phase terni (o from the expression. However, this term is not directly accessible 
from a single image. In equation (6) there are three unknown values and the form of the equation is quite simple. As 
a result, mathematically only three samples (from three images) are required to retrieve an estimate of the phase term, 
which is equivalent to the distance of an object in the scene from the imaging system 10. Therefore, a set of three 
^5 images captured with unique phase shifts is sufficient to determine co. For simplicity, the phase shifts are given by 0^ 
2nk/Z\ 0,1 ,2. In the following description, an image bundle shall be understood to constitute a collection of Images 
which are of the same scene, but with each image having a distinct phase offset obtained from the modulation applied 
to the intensifier 26. It should also be understood that an analogous analysis may be performed by phase shifting the 
illuminator 14 instead of the intensifier 26. If an image bundle comprising more than three images is captured, then 
the estimates of range can be enhanced by a least squares analysis using a singular value decomposition (see, e.g., 
W.H. Press, B.P. Flannery, S.A. Teukolsky and W.T Vetterling, Numerical Recipes (the Art of Scientific Computing), 
Cambridge University Press, Cambridge, 1986). 

[0027] If images are captured with n >3 distinct phase offsets of the intensifier (or laser or a combination of both) 
these images form an image bundle. Applying Equation (6) to each image in the image bundle and expanding the 
cosine term (i.e., Pj =2|iijiM7c + K7ry(cos((o)cos(0|)-sln((t>)sln(6i))) results In the following system of linear equations In n 
unknowns at each point: 



(Eq. 7) 

where A = 2hl^m^» ^2 = »c7rycos(o, and A3=K7rysinco. This system of equations is solved by a singular value decomposition 
to yield the vector A = [A^ , A2, As]^. Since this calculation Is carried out at every (x,y) location in the Image bundle, A 
Is really a vector image containing a three element vector at every point. The phase term a> is computed at each point 
using a four-quadrant arctangent calculation: 



40 



45 





r 


p. 






V 



1 cos^, 
1 cos^. 



-sin^, ^ 
- sin 

' sin 0^ 



A. 

1^3 j 



55 



(o = tan"^ (A3, A2) (Eq. 8) 

The resulting collection of phase values at each point forms the phase image. Once phase has been determined, range 
rcan be calculated by : 
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r=«>^ (Eq.9) 

Equations (1 ) - (9) thus describe a method of estimating range using an image bundle with at least three images (i.e.. 

5 n = 3) corresponding to distinct phase offsets of the intensifier and/or laser. 

[0028] Referring now to Figure 1^ the overall color scannerless range imaging system is shown to comprise four 
main components in accordance with the present Invention. The first component is an image capture device 40 for 
capturing an image. This may be, for example, a camera body where an image capture element 42 might be either a 
photosensitive film or an electronic sensor, such as a charged-coupled-device. This device 40 Is able to capture a 

10 plurality of images. Moreover, a remote trigger line 82 is provided for signaling the image capture device 40. Also, a 
means for advancing, or driving, the image capture device 40 to prepare for a successive Image capture must be 
available. Such capabilities are welt known in relation to such image capture devices, and will not be described in 
detail. The second component is an illumination system 50 for producing high-frequency amplitude modulated illumi- 
nation of a desired average amplitude, amplitude modulus and frequency. It is also desirable that the illumination 

'5 system 50 further includes the capability of shifting the phase of the amplitude modulation between a set of prescribed 
phase offsets (alternatively, this function may be performed by modulation of the reflected illumination in the capture 
portion of the color scannerless range imaging system). It is also useful for the Illumination system 50 to have a preferred 
operating wavelength. The third component is an optical assembly 60 comprised of two optical elements 62 and 64 
and a component 66 (such as, will be seen, a lens turret) for interchanging the optical elements 62 and 64 In front of 

^0 the image capture device 40. The two optical elements 62 and 64 have different purposes, but both share the same 
gross optical properties, such as focal length, aperture and field of view. A controller 80 manages the overall image 
capture process of the image and range capture system, including the operation of the Illumination system 50, the 
image capture device 40, and the optical assembly 60. For instance, the controller 80 may include the remote trigger 
line 82 for causing the image capture device 40 to initiate an image capture sequence, and an illumination control line 

^5 84 for causing the illumination system 50 to emit the correct illumination at the connect moment. It is preferred, but not 
mandatory, that the controller 80 automatically operates the component 66, via an optics control line 86, to change the 
optical elements 62 and 64 as needed for operation according to the Invention. 

[0029] As shown in relation to Figure 2, the notion of an Image bundle 90 Is central to the range estimation method. 
The image bundle 90 includes a combination of images captured by the system as well as infomnatlon pertinent to the 

30 individual images and information common to all the images. The Image bundle contains two types of images: range 
images 92 related to the range capture portion of the process and a color image 94, commonly referred to as a texture 
image. Common information 96 in the image bundle 90 would typically include the number of images in the bundle 
(three or more) and the modulation frequency utilized by the camera system. Other information might be the number 
of horizontal and vertical pixels in the images and/or data related to camera status at the time of the image capture. 

55 Image specific information will Include the phase offset 1 ...N used for each (1 ...N) of the individual range Images 92. 
The Image bundle 90 includes a minimum of three such Images, each of which are monochrome. Each of the range 
images 92 records the effect of a distinct phase offset applied to either the illumination system 50 or to elements of 
the optical assembly 60. The additional color image 94 is an image using the optical element 62 that does not contain 
range capture components. Although this Is a color image, it Is preferably, but not necessarily, the same size as the 

40 range Images 92. 

[0030] The Image capture device 40 shown in Figure 3 shares many aspects of a standard camera body (except 
that, in this embodiment, image capture is initiated from the controller 80). For the purpose of the present invention, 
the image capture element 42 is enabled to capture color images. In the case of a digital image capture device, the 
capture element 42 (e.g., a charge-coupled device) is covered with a color-filter-array (not shown) for generating a 

45 color image, and in the case of a film image capture system a color film is used and a method of introducing fiducial 
markings on the film is preferred for registration. The image capture system requires image storage means to store all 
range images 92 in the image bundle 50, as well as a color texture image 94 in addition to the range images 92. This 
can be accomplished by an on-camera storage means, such as a photosensitive film 44 with multiple image frames 
or a digital storage mechanism 46, such as an internal memory together with output connections to, e.g., a PCMCIA 

50 card or a floppy disk (not shown) for receiving images from the internal memory. A camera controller 48 connected to 
the remote trigger line 82 accepts trigger signals and causes the image capture device 40 to capture an image. Once 
an image is recorded the image capture device 40 must automatically prepare for an additional image capture. In the 
case of a film based image capture system, an automatic film advance (not shown) is activated by the camera controller 
48. In the case of a digital camera, the camera controller 48 stores the image bundle data onto the digital storage 

55 mechanism 46 and clears internal buffers (not shown) for a subsequent image capture. 

[0031] The illumination system 60 shown in Figure 4 has the primary purpose of producing amplitude-modulated 
illumination with its phase controllable for generating a shift in the transmitted wave pattem for each range Image in 
the image bundle (although, as mentioned before, this function may be performed by modulation of the reflected illu- 
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minatlon in the capture portion of the color scannerless range imaging systenn). The illumination system 50 includes 
a light source, which Is preferably a laser light source 52 with an intensity of about 0.5 watt, and a modulation circuit 
54 controllable through line 84 from the controller 80, for generating the requisite modulation signals of predetermined 
frequency with a set of predetermined phase offsets. The laser light source 52 is preferably modulated at a modulation 
5 frequency of about 12.5 megahertz and the preferred phase offsets, as mentioned earlier, are phase shifts e in each 
range image given by 6^^ = 27c/c/3; k= 0,1 ,2. The preferred wavelength of the laser light is about 830 nm, as this wave- 
length provides an optimal balance between concerns for eye-safety and for the typical response of one or more of 
the optical elements 62 and 64 described below. Although the laser light need not necessarily be uniformly distributed, 
a diffusion lens 56 is positioned in front of the laser light source 52 in order to spread the modulated light across the 

10 desired field of view as uniformly as possible. 

[0032] The illumination system 50 also includes a standard wide-band illumination system 58 that is not modulated. 
This illumination source is used for nomnal photographic images. This device 58 may be a commonly known and 
understood flash of a standard camera system, e.g., a commonly available electronic flash of the type useful with 
photographic cameras. The illumination system 50 is connected via the control line 84 to the controller 80. which directs 

15 the illumination system 50 to operate in either of the following modes: a) a first mode in which the laser is operated to 
illuminate the scene with a plurality (bundle) of exposures, each with a unique phase offset applied to its modulating 
frequency; and b) a second mode in which the standard wide-band Illumination system 58 is turned on and the flash 
is Initiated by the controller during capture of the color texture Image. If ambient light is sufficient, of course, it may be 
unnecessary for the illumination system 50 to operate in the second mode in order to capture a color image; in that 

20 case, the image capture device would be instructed to operate without flash. Moreover, the sequence of image capture 
may be reversed, that is, the second mode may be engaged before the first mode or, indeed, the second mode might 
in specific situations be engaged between the several exposures of the first mode. The illumination system 50 also 
communicates with the controller 40 via the line 84 to indicate that all systems are ready for use. 
[0033] Referring to Figure 5, the optical assembly 60 includes a lens turret 66 for supporting the primary optical 

25 elements 62 and 64 in relation to the image capture device 40. (The optical elements 62 and 64 are shown in Figure 
5 removed from their threaded lens openings 62a and 64a in the lens turret 66.) The turret 66 is movable about a pivot 
68 interconnected with the image capture device 40 for placing a selected one of the optical elements 62 and 64 in an 
optical path of the image capture element 42. These optical elements 62 and 64 share as many optical properties as 
practicable, such as focal length, distortion, field of view, and aperture. The first optical element 62 is a standard optical 

30 element as conventionally found on a large number of camera systems for imaging a scene on the image capture 
element 42. The alternative optical element 64 contains components to enable the range capture, thereby imaging the 
bundle of phase interference pattems on the image capture element 42. Such components would include the image 
intensifier 26 shown in Figure 8 (including the photocathode 24, the microchannel 30 and the phosphor screen 32), as 
well as a power supply 70 and modulation control circuitry 72 (which may be a part of the modulator 1 6 shown in Figure 

35 8). The intensifier 26 is triggered by an Intensifier control line 85 connected to the controller 80, which drives the power 
supply 70 and cycles the modulation control circuitry 72 for each range image capture. As alluded to earlier, the mod- 
ulation control circuitry 72 in the optical element 64 may optionally produce amplitude-modulated illumination with its 
phase controllable for generating the before-mentioned shift (i.e., with preferred phase shifts, as mentioned earlier, 
given by 6;^ = 2nkfZ\ k= 0,1 ,2) in the transmitted wave pattern for each range image in the image bundle . 

40 [0034] The lens turret 66 in the optical assembly 60 permits either of the two optical elements 62 and 64 to be placed 
in line with the optical axis of the image capture device 40. Such a turret housing is well known and its construction 
will not be described in detail, e.g., optical systems having multiple conventional lenses on a turret are well known in 
the motion picture industry. For instance, the disclosure of U.S. Patent No. 4,307,940, which is incorporated herein by 
reference, is an example of such a system in a reproduction camera. These systems permit the operator to easily shift 

45 between a plurality of lenses without having to detach and re-attach a lens. Advantages of such a system is the ease 
and speed of interchanging optical elements, as well as maintaining the position of the camera during the Interchange 
of optical elements. It is preferable, but not required^ that the ability to pivot the turret 66 be accomplished automatcally 
without the need of the operator to physically cause the elements to be interchanged. In such case, a motor 74 is 
operated by direction of the controller 80 for rotating the turret 66 between its two optical positions. Given the position 

50 of the motor shaft, the optical assembly 60 signals the controller 80 via the optics control line 86 regarding the current 
position of the optical assembly 60. 

[0035] Referring to Figure 6, the main purpose of the controller 80 is in sequencing and othenvise controlling the 
operation of the components of the color scanneriess range imaging system to capture the image bundle 90 shown in 
Figure 2. The controller 80 is shown to include a processor 87, a switch 88 and an output interface 89. More particularly, 
55 the processor 87 determines the current state of image collection to complete an image bundle 90 and, if another 
image is needed, establishes the parameters for the next successive image in the image bundle. The processor 87 
communicates via the interface 89 and the remote trigger line 82 with the image capture device 40 to activate the 
image capture element 42 and capture an image. This function is initiated by the switch 88 in the controller 80. which 
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connects a user controlled trigger signal to the processor 87. The controller 80 also communicates via the interface 
89 with the illumination system 50 through the control line 84. During all of this activity, the controller 80 maintains the 
capture status of the image bundle 90. If a range image 92 contained in the image bundle 90 is set to be captured, 
then the controller 80 signals the optical assembly 60 via the optics control line 86 to align the optical element 64 with 
5 the optical path of the image capture device 40. If the range images 92 in the bundle 90 are complete and a color 
texture image 94 is required, then the controller 80 signals the optical assembly 60 to position itself such that alternative 
optical element 62 is in the optical path of the image capture device 40. 

[0036] The processing of the image bundle 90 is described in Figure 7. The range images 92 of the bundle 90 and 
the color texture Image 94 are combined to form a color Image with a dense set of range values, that is, where range 

10 values are associated with a preponderance of, or all. image pixel locations. The range images 92 contained in the 
image bundle 90 are used to detennine the range estimates for each pixel in the image. The process begins with the 
opening of the image bundle (SI 00), and the Initializing of internal parameters, including the number of range images, 
the number of pixels to be processed and the modulation frequency of the camera system. A processing loop Is then 
entered that operates on the image In a pixel-wise fashion. After the range map is opened (S102), the first step is to 

'5 estimate the range (S104) for the (ij)th pixels. For each pixel location (i,j), the range is estimated by sampling each 
range image at its (ij)th pixel location and perfomning the aforementioned calculation. The pixel intensity values and 
the phase offset used in producing the Image In the Image bundle are matched. The aforementioned equations (1) - 
(9) describe a method of estimating range using an image bundle with at least three images corresponding to distinct 
phase offsets of the intensifier and/or laser. Moreover, it is well known that there is a sinusoidal relationship between 

20 the intensity values and the phase offset. By fitting the data to a sine-wave of the form a + p sin((p + co) then the range 
can be estimated. As a by-product of this estimate, an estimate of the luminance value (S1 06) of the pixel is obtained. 
By performing these operations the estimated luminance value and the estimated range at the (i,j)th pixel Is obtained 
and stored (SI 08). The pixel location Is incremented (S1 10) and if all pixels are processed (S112) then the loop is 
exited. Otherwise the next pixel is processed in the same manner 

25 [0037] A full color image 94 is acquired when the alternative optical element 62 is aligned with the optical path of the 
image capture device 40. The luminance image estimated above and the color image need to be correlated (8114). 
There are many approaches to accomplish this task, e.g.. refer to The Image Processing Handbook, 2"^ ed., by John 
C. Russ. CRC Press. 1995, pp. 332-336. These two images are then correlated (S114) in order to eliminate any dis- 
tortions caused by subtle differences in the physical optics of the two lenses comprising the optical assembly 80. A 

30 warping of the luminance image (8116) to match the features of the color image is obtained. The warping is applied 
to the range data (8118) in order to match the range data with the color image features. Such warping techniques are 
described in Digital Image Warping, by G. Wolberg, IEEE Computer Society Press, 1 990, The warped range image 
and the color Image are combined and stored as a color image with range data (S 120). It Is pretended that the range 
data be stored as floating point values. 

55 [0038] The aforementioned processing of the image bundles 90 is preferably performed on any well-known computer 
system, such as the computer system shown in Figure 9. It is also instructive to note that the images may be either 
directly Input into the computer system from the color scannerless range imaging system (particularly if It is a digital 
capture system) or digitized before input into the computer system (for example by scanning an original, such as a 
silver halide film, if the imaging system is a film system). Referring to Fig. 9, there is illustrated a computer system 110 

40 for implementing the programs associated with the present invention. Although the computer system 110 is shown for 
the purpose of illustrating a preferred embodiment, the present invention is not limited to the computer system 110 
shown, but may be used on any electronic processing system. The computer system 110 Includes a microprocessor- 
based unit 112 for receiving and processing software programs of the type illustrated In Figure 7 and for perfomning 
other processing functions. A display 11 4 is electrically connected to the microprocessor-based unit 1 12 for displaying 

45 user-related Information associated with the software, e.g., by means of a graphical user interface. A keyboard 11 6 Is 
also connected to the microprocessor based unit 112 for permitting a user to Input infonnation to the software. As an 
alternative to using the keyboard 11 6 for input, a mouse 1 1 8 may be used for moving a selector 120 on the display 114 
and for selecting an item on which the selector 120 overiays, as Is well known in the art. 

[0039] A compact disk-read only memory (CD-ROM) 122 is connected to the microprocessor based unit 112 for 
50 receiving software programs and for providing a means of inputting the software programs and other infomnation to 
the microprocessor based unit 112 via a compact disk 124, which typically includes a software program. In addition, a 
floppy disk 1 26 may also include a software program, and is inserted into the microprocessor-based unit 1 12 for inputting 
the software program. Still further, the microprocessor-based unit 112 may be programmed, as is well known in the 
art, for storing the software program internally. The microprocessor-based unit 1 1 2 may also have a network connection 
55 127, such as a telephone line, to an external network, such as a local area network or the Internet A printer 128 Is 
connected to the microprocessor-based unit 112 for printing a hardcopy of the output of the computer system 110. 
[0040] Images may also be displayed on the display 14 via a personal computer card (PC card) 130, such as, as it 
was fomneriy known, a PCMCIA card (based on the specifications of the Personal Computer Memory Card International 
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Association) which contains digitized innages from the color scanneriess range imaging system electronically embodied 
in the card 130. The PC card 130 is ultimately inserted into the microprocessor based unit 112 for pemnitting visual 
display of the image on the display 114. Images may also be input via the compact disk 124, the floppy disk 126, or 
the network connection 127. After the images have been processed in accordance with the steps shown in Figure 7. 
5 the range data is stored together with color values, such as RGB data, for each image pixel. Such stored data may be 
accumulated for the entire image and wrinen as a dense range map including color data onto the compact disk 124, 
the floppy disk 126, or the card 130, or communicated via the network connection 127. 

[0041] These and other aspects, objects, features and advantages of the present invention will be more clearly 
understood and appreciated from a review of the following detailed description of the preferred embodiments and 
10 appended claims, and by reference to the accompanying drawings. 

Claims 

IS 1 . A color scanneriess range imaging system for generating Images of a scene, said system comprising; 

a illumination system for controllably Illuminating the scene with modulated illumination, whereby some of the 
modulated illumination Is reflected from objects In the scene; and 

^o an image capture device positioned in an optical path of the reflected illumination from the scene for capturing 

a plurality of images thereof, including (a) at least one image of the reflected modulated illumination, whereby 
the modulation of the reflected modulated Illumination incorporates a phase delay corresponding to the dis- 
tance of objects in the scene from the range imaging system, and (b) at least one color image of reflected 
unmodulated illumination. 

25 

2. The system as claimed in claim 1 further comprising means for storing the plurality of images as a bundle of 
associated Images. 

3. The system as claimed in claim 1 wherein the image capture system captures a plurality of images of the reflected 
30 modulated illumination, wherein each image incorporates the effect of a predetemnined modulation frequency to- 
gether with a phase offset unique for each image. 

4. The system as claimed in claim 3 wherein each unique phase offset 0 is given by = 2Ki/3; i= 0,1 ,2. 

35 5. The system as claimed in claim 1 wherein the bundle of images is stored on a photographic film. 

6. The system as claimed in claim 1 wherein the bundle of Images include digital images stored on digital storage 
media. 

40 7. The system as claimed in claim 1 wherein the illumination system includes a laser illuminator for producing the 
modulated illumination. 

8. A color scanneriess range Imaging system for generating Images of a scene, said system comprising; 

^s an illumination system for illuminating the scene with modulated illumination, whereby illumination is reflected 

from objects in the scene; 

an image capture device positioned in an optical path of the reflected illumination from the scene for capturing 
a plurality of images thereof, including (a) at least one image of the reflected modulated illumination, whereby 
so the modulation of the reflected modulated illumination incorporates a phase delay corresponding to the dis- 

tance of objects in the scene from the range imaging system, and (b) at least one color image of reflected 
unmodulated illumination; 

an optical assembly including a first optical system for imaging a color image of unmodulated illumination from 
ss the scene and a second optical system including a modulating stage for modulating the reflected modulated 

illumination from the scene and generating an interference image representative of range infomiation of the 
scene; and 
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a controller for selectively placing each of the optical systems in the optical path of the image capture device 
whereby both range information and color Information of the scene are captured by the image capture device. 

9. The system as claimed in claim 8 wherein the illumination system additionally produces unmodulated illumination 
5 for illuminating the scene and producing at least in part the reflected unmodulated illumination captured by the 

image capture device as the color image. 

10. The system as claimed In claim 8 wherein the modulated Illumination is amplitude modulated by a modulating 
signal of a predetermined modulating frequency. 

10 
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